Training neural networks of an automatic clinical workflow that recognizes and analyzes 2D and doppler modality echocardiogram images

ABSTRACT

A method for training neural networks of an automated workflow performed by a software component executing on a server in communication with remote computers at respective laboratories includes downloading and installing a client and a set of neural networks to a first remote computer of a first laboratory, the client accessing the echocardiogram image files of the first laboratory to train the set of neural networks and to upload a first trained set of neural networks to the server. The process continues until the client and the second trained set of neural networks is downloaded and installed to a last remote computer of a last laboratory, the client accessing the echocardiogram image files of the last laboratory to continue to train the second trained set of neural networks and to upload a final trained set of neural networks to the server.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of co-pending patentapplication Ser. No. 16/216,929, filed Dec. 11, 2018, assigned to theassignee of the present application and incorporated herein byreference.

BACKGROUND

The present invention relates to image classification for diseaseprediction, and more specifically, to training neural networks of anautomatic clinical workflow that recognizes and analyzes 2D and dopplermodality echocardiogram images for automated measurements and thediagnosis, prediction and prognosis of heart disease.

Cardiovascular disease including heart failure is a major health problemaccounting for about 30% of human deaths worldwide. Heart failure isalso the leading cause of hospitalization in adults over the age of 65years. Echocardiography is an important diagnostic aid in cardiology forthe morphological and functional assessment of the heart. In a typicalpatient echocardiogram (echo) examination, a clinician called asonographer places an ultrasound device against the patient's chest tocapture a number of 2D images/videos of the patients' heart. Reflectedsound waves reveal the inner structure of the heart walls and thevelocities of blood flows. The ultrasound device position is variedduring an echo exam to capture different anatomical sections as 2Dslices of the heart from different viewpoints or views. The clinicianhas the option of adding to these 2D images a waveform captured fromvarious possible modalities including continuous wave Doppler, m-mode,pulsed wave Doppler and pulsed wave tissue Doppler. The 2D images/videosand Doppler modality images are typically saved in DICOM (DigitalImaging and Communications in Medicine) formatted files. Although thetype of modality is partially indicated in the metadata of the DICOMfile, the ultrasound device position in both the modality and 2D views,which is the final determinant of which cardiac structure has beenimaged, is not.

After the patient examination, a clinician/technician goes through theDICOM files, manually annotates heart chambers and structures like theleft ventricle (LV) and takes measurements of those structures. Theprocess is reliant on the clinicians' training to recognize the view ineach image and make the appropriate measurements. In a follow upexamination, a cardiologist reviews the DICOM images and measurements,compares them to memorized guideline values and make a diagnosis basedon the interpretation made from the echocardiogram.

The current workflow process for analyzing DICOM images, measuringcardiac structures in the images and determining, predicting andprognosticating heart disease is highly manual, time-consuming anderror-prone. Because the workflow process is so labor intensive, morethan 95% of the images available in a typical patient echocardiographicstudy are never annotated or quantified. The view angle or Dopplermodality type by which an image was captured is typically not labelled,which means the overwhelming majority of stored DICOMs from past patientstudies and clinical trials do not possess the basic structure andnecessary identification of labels to allow for machine learning on thisdata.

There has been a recent proposal for automated cardiac imageinterpretation to enable low cost assessment of cardiac function bynon-experts. Although the proposed automated system holds the promise ofimproved performance compared to the manual process, the system hasseveral shortfalls. One shortfall is that the system only recognizes 2Dimages. In addition, although the proposed system may distinguishbetween a normal heart and a diseased heart, the proposed system isincapable of distinguishing hearts having similar-looking diseases.Consequently, the number of heart diseases identified by the proposedsystem is very limited and requires manual intervention to identifyother types of heart diseases.

For example, heart failure has been traditionally viewed as a failure ofcontractile function and left ventricular ejection fraction (LVEF) hasbeen widely used to define systolic function, assess prognosis andselect patients for therapeutic interventions. However, it is recognizedthat heart failure can occur in the presence of normal or near-normalEF: so-called “heart failure with preserved ejection fraction (HFPEF)”which accounts for a substantial proportion of clinical cases of heartfailure. Heart failure with severe dilation and/or markedly reduced EF:so-called “heart failure with reduced ejection fraction (HFREF)” is thebest understood type of heart failure in terms of pathophysiology andtreatment. The symptoms of heart failure may develop suddenly ‘acuteheart failure’ leading to hospital admission, but they can also developgradually. Timely diagnosis, categorization of heart failuresubtype-HFREF or HFPEF, and improved risk stratification are criticalfor the management and treatment of heart failure, but the proposedsystem does not address this.

The proposed system is also incapable of generating a prognosis based onthe identified heart disease, and would instead require a cardiologistto manually form the prognosis. The proposed system is further incapableof structuring the automated measurements and labelled views acrossmultiple sources of data, to enable training and validation of diseaseprediction algorithms across multiple remote patient cohorts.

Accordingly, there is a need for an improved and fully automaticclinical workflow that recognizes and analyzes both 2D and Dopplermodality echocardiographic images and a method of training the neuralnetworks used by the automatic clinical workflow.

BRIEF SUMMARY

The exemplary embodiments provide methods and systems for trainingneural networks of an automated workflow performed by a softwarecomponent executing on a server in communication with remote computersat respective laboratories. Aspects of exemplary embodiment includedownloading and installing a client and a set of neural networks to afirst remote computer of a first laboratory, the client accesses theechocardiogram image files of the first laboratory to train the set ofneural networks and to upload a first trained set of neural networks tothe server. The client and the first trained set of neural networks isdownloaded and installed to a second remote computer of a secondlaboratory, the client accesses the echocardiogram image files of thesecond laboratory to continue to train the first trained set of neuralnetworks and to upload a second trained set of neural networks to theserver. The process continues until the client and the second trainedset of neural networks is downloaded and installed to a last remotecomputer of a last laboratory, the client accesses the echocardiogramimage files of the last laboratory to continue to train the secondtrained set of neural networks and to upload a final trained set ofneural networks to the server. In a further embodiment, the softwarecomponent executing on the server uses the final trained set of neuralnetworks in an analysis mode to automatically recognize and analyze theechocardiogram images in the patient studies of the respective labs.

According to the method and system disclosed herein, the disclosedembodiments provides a federated training platform that accessesstructured echo image databases in multiple lab locations to allowdistributed neural network training and validation of disease predictionacross multiple patient cohorts without either the original echo imagesfiles or the labelled echo image files stored in the structured databaseever having to leave the premise of the labs.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIGS. 1A-1C are diagrams illustrating embodiments of a system forimplementing an automated clinical workflow that recognizes and analyzesboth 2D and Doppler modality Echocardiographic images for automatedmeasurements and the diagnosis, prediction and prognosis of heartdisease.

FIG. 2 illustrates architectural layers of the echo workflow engine.

FIG. 3 is a flow diagram illustrating one embodiment of a process forperformed by the echo workflow engine to automatically recognize andanalyze both 2D and Doppler modality echo images to perform automatedmeasurements and the diagnosis, prediction and prognosis of heartdisease.

FIG. 4A is a flow diagram illustrating details of the process forautomatically recognizing and analyze both 2D and Doppler modality echoimages to perform automated measurements and the diagnosis, predictionand prognosis of heart disease according to one embodiment.

FIG. 4B is a flow diagram illustrating advanced functions of the echoworkflow engine.

FIG. 5A is diagram illustrating an example 2D echo image.

FIG. 5B is diagram illustrating an example Doppler modality image.

FIGS. 6A-6K are diagrams illustrating some example view typesautomatically classified by the echo workflow engine.

FIG. 7 is a diagram illustrating an example 2D image segmented toproduce annotations indicating cardiac chambers, and an example Dopplermodality image segmented to produce a mask and trace waveform

FIGS. 8A and 8B are diagrams illustrating examples of findingsystolic/diastolic end points.

FIGS. 9A and 9B are diagrams illustrating processing of an imagingwindow in a 2D echo image and the automated detection of out of sectorannotations.

FIGS. 10A and 10B are diagrams graphically illustrating structuralmeasurements automatically generated from annotations of cardiacchambers in a 2D image, and velocity measurements automaticallygenerated from annotations of waveforms in a Doppler modality.

FIG. 11 is a diagram graphically illustrating measurements of globallongitudinal strain that were automatically generated from theannotations of cardiac chambers in 2D images.

FIG. 12A is a diagram graphically illustrating an example set of bestmeasurement data based on largest volume cardiac chambers and the savingof the best measurement data to a repository.

FIG. 12B is a diagram graphically illustrating the input ofautomatically derived measurements from a patient with normal LV EFmeasurements into a set of rules to determine a conclusion that thepatient has normal diastolic function, diastolic dysfunction, orindeterminate.

FIG. 13 is a diagram graphically illustrating the output ofclassification, annotation and measurement data to an example JSON file.

FIG. 14 is a diagram illustrating a portion of an example report showinghighlighting values that are outside the range of Internationalguidelines.

FIG. 15 is a diagram illustrating a portion of an example report of MainFindings that may be printed and/or displayed by the user.

FIG. 16A is a diagram showing a graph A plotting PCWP and HFpEF scores.

FIG. 16B is a diagram showing a graph B plotting CV mortality orhospitalization for HF.

FIG. 17 is a diagram illustrating a federated training platformassociated with the automated clinical workflow system.

FIG. 18 is a diagram showing use of a GAN for creating imagesegmentation training data.

FIG. 19 is a diagram showing components of a GAN for creating viewclassification training data.

DETAILED DESCRIPTION

The exemplary embodiments relate to training neural networks of anautomatic clinical workflow that recognizes and analyzes 2D and Dopplermodality Echocardiographic images. The following description ispresented to enable one of ordinary skill in the art to make and use theinvention and is provided in the context of a patent application and itsrequirements. Various modifications to the exemplary embodiments and thegeneric principles and features described herein will be readilyapparent. The exemplary embodiments are mainly described in terms ofparticular methods and systems provided in particular implementations.However, the methods and systems will operate effectively in otherimplementations. Phrases such as “exemplary embodiment”, “oneembodiment” and “another embodiment” may refer to the same or differentembodiments. The embodiments will be described with respect to systemsand/or devices having certain components. However, the systems and/ordevices may include more or less components than those shown, andvariations in the arrangement and type of the components may be madewithout departing from the scope of the invention. The exemplaryembodiments will also be described in the context of particular methodshaving certain steps. However, the method and system operate effectivelyfor other methods having different and/or additional steps and steps indifferent orders that are not inconsistent with the exemplaryembodiments. Thus, the present invention is not intended to be limitedto the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features described herein.

The disclosed embodiments provide method and system for training andimplementing a software-based automatic clinical workflow thatrecognizes and analyzes both 2D and Doppler modality Echocardiographicimages for automated measurements and the diagnosis, prediction andprognosis of heart disease, and which can be deployed in workstation ormobile-based ultrasound point-of-care systems.

FIGS. 1A-1C are diagrams illustrating embodiments of a system forimplementing an automated clinical workflow that recognizes and analyzesboth 2D and Doppler modality Echocardiographic images for automatedmeasurements and the diagnosis, prediction and prognosis of heartdisease. FIG. 1A shows a basic standalone configuration for theautomated clinical workflow system 10A. The automated clinical workflow10A is primarily implemented as a software application, referred to asecho workflow engine 12, that executes on a computer 14 operating in astandalone setting, disconnected from other devices on network 26. Thecomputer 14 may be implemented in any form factor including aworkstation, desktop, notebook, laptop server or tablet capable ofrunning an operating system, such as Microsoft Windows® (e.g., Windows7®, Windows 10®), Apple macOS®, Linux®, Apple iOS®, Android®, and thelike.

The computer 14 may include typical hardware components (not shown)including a processor, input devices (e.g., keyboard, pointing device,microphone for voice commands, buttons, touchscreen, etc.), outputdevices (e.g., a display device, speakers, and the like), and wired orwireless network communication interfaces (not shown) for communication.The computer 14 may include internal computer-readable media, such asmemory (not shown) containing computer instructions comprising the echoworkflow engine 12, which implements the functionality disclosed hereinwhen executed by one or more computer processors.

The computer 14 may further include local internal storage for storingone or more databases 16 and an image file archive 18. In oneembodiment, the contents of the image file archive 18 includeechocardiogram image files (also referred to herein as echo images),which in some embodiments may be stored in DICOM (Digital Imaging andCommunications in Medicine) format.

In one embodiment, the computer 14 is in communication with peripheraldevices such an ultrasound imaging device 24 that capturesechocardiogram images of a patient's organ (e.g., a heart), which maythen be stored as a patient study using the database 16 and image filearchive 18. For example, the computer 14 may be located in a hospital orclinical lab environment where Echocardiography is performed as adiagnostic aid in cardiology for the morphological and functionalassessment of the heart. During a typical patient echocardiogram exam(referred to as a study), a sonographer or technician places theultrasound imaging device 24 against the patient's chest to capture 2Decho images/videos of the heart to help diagnose the particular heartailment. Measurements of the structure and blood flows are typicallymade using 2D slices of the heart and the position of the ultrasoundimaging device 24 is varied during an echo exam to capture differentanatomical sections of the heart from different viewpoints. Thetechnician has the option of adding to these 2D echo images a waveformcaptured from various possible modalities including: continuous waveDoppler, m-mode, pulsed wave Doppler and pulsed wave tissue Doppler. The2D images and Doppler waveform images may be saved as DICOM files.Although the type of modality is sometimes indicated in the metadata ofthe DICOM file, the 2D view is not.

The computer 14 may further include removable storage devices such as anoptical disk 20 and/or a flash memory 22 and the like for storage of theecho images. In some embodiments, the removable storage devices may beused as an import source of echo images and related data structures intothe internal image file archive 18, rather than or in addition to, theultrasound imaging device 24. The removable storage devices may also beused as an archive for echocardiogram data stored in the database 16and/or the image file archive 18.

In an advanced embodiment referred to as connected configuration 10B,the computer 14 may be connected through the network 26 and router 28 toother DICOM based devices, such as DICOM servers 30, network file sharedevices 32, Echo workstations 34, and/or cloud storage services 36hosting DICOM files. In the connected configuration 10B, severalpossible interactions with the database 16 and the image file archive 18are possible, as described below.

One possible interaction is to use the cloud storage services 36 as aninternal archive. In case of very large archives consisting of largeamounts of DICOM files, the computer 14 may not have sufficient storageto host all files and the echo workflow engine 12 may be configured touse external network storage of the cloud storage services 36 for filestorage.

Another possible interaction is to use the cloud storage services 36 asan import source by i) selecting a DICOM file set or patient study,which includes the DICOM and Doppler waveforms images and patient dataand examination information. The patient study may also be selected by areserved DICOMDIR file instance, from which the patient, exams and imagefiles are read.

Yet a further possible interaction is to use the DICOM servers 30, thenetwork file share devices 32, echo workstations 34, and/or DICOMclients (of FIG. 1C) acting as DICOM servers (workstations withmodalities CFind, CMove and CStore) in order to retrieve patients, examsand images by performing a CFind operation, followed by a CMoveoperation, to request the remote device to send the images resultingfrom the CFind operation.

Referring now to FIG. 1B, a handheld configuration 10C for the automatedclinical workflow system is shown. In this embodiment, the computer 14of FIG. 1A is implemented as a handheld device 14′, such as a tablet ora mobile phone, connected to a wired or wireless portable ultrasoundscanner probe 24′ that transmits echo images to the handheld device 14′.In such embodiments, the echo workflow engine 12 may be implemented asan application executed by the handheld device 14′.

FIG. 1C illustrates a software as a service (SaaS) configuration 10D forthe automated clinical workflow. In this embodiment, the echo workflowengine 12 is run on a server 40 that is in communication over thenetwork 26 with a plurality of client devices 42. In this embodiment,the server 40 and the echo workflow engine 12 may be part of athird-party service that provides automated measurements and thediagnosis, prediction and prognosis of heart disease to client devices(e.g., hospital, clinic, or doctor computers) over the Internet. Itshould be understood that although the server 40 is shown as a singlecomputer, it should be understood that the functions of server 40 may bedistributed over more than one server. In an alternative embodiment, theserver 40 and the echo workflow engine 12 of FIG. 1C may be implementedas a virtual entity whose functions are distributed over multiple clientdevices 42. Likewise, it should be understood that although the echoworkflow engine 12 is shown as a single component in each embodiment,the functionality of the echo workflow engine 12 may be separated into agreater number of modules/components.

Conventionally, after a patient examination where echo images arecaptured stored, a clinician/technician goes through the DICOM files,manually annotates heart chambers and structures and takes measurements,which are presented in a report. In a follow up examination, a doctorwill review the DICOM images and measurements, compare them to memorizedguideline values and make a diagnosis. Such a process is reliant on theclinicians' training to recognize the view and make the appropriatemeasurements so that a proper diagnosis can be made. Such a process iserror-prone and time consuming.

According to the disclosed embodiments, the echo workflow engine 12mimics the standard clinical practice, processing DICOM files of apatient by using a combination of machine learning, image processing,and DICOM workflow techniques to derive clinical measurements, diagnosespecific diseases, and prognosticate patient outcomes, as describedbelow. While an automated solution to echo image interpretation usingmachine learning has been previously proposed, the solution onlyanalyzes 2D echo images and not Doppler modality waveform images. Thesolution also mentions disease prediction, but only attempts to handletwo diseases (hypertrophic cardiomyopathy and cardiac amyloidosis) andthe control only compares normal patients to diseased patients.

The echo workflow engine 12 of the disclosed embodiments, however,improves on the automated solution by utilizing machine learning toautomatically recognize and analyze not only 2D echo images but alsoDoppler modality waveform images. The echo workflow engine 12 is alsocapable of comparing patients having similar-looking heart diseases(rather than comparing normal patients to diseased patients), andautomatically identifies additional diseases, including both heartfailure with reduced ejection fraction (HFrEF) and heart failure withpreserved ejection fraction (HFpEF). HFrEF is known as heart failure dueto left ventricular systolic dysfunction or systolic heart failure andoccurs when the ejection fraction is less than 40%. HFpEF is a form ofcongestive heart failure where in the amount of blood pumped from theheart's left ventricle with each beat (ejection fraction) is greaterthan 50%. Finally, unlike the proposed automated solution, the echoworkflow engine 12 automatically generates a report with the results ofthe analysis for medical decision support.

FIG. 2 illustrates architectural layers of the echo workflow engine 12.In one embodiment, the echo workflow engine 12 may include a number ofsoftware components such as software servers or services that arepackaged together in one software application. For example, the echoworkflow engine 12 may include a machine learning layer 200, apresentation layer 202, and a database layer 204.

The machine learning layer 200 comprises several neural networks toprocess incoming echo images and corresponding metadata. The neuralnetworks used in the machine learning layer may comprise a mixture ofdifferent classes or model types. In one embodiment, machine learninglayer 200 utilizes a first neural network to classify 2D images by viewtype, and uses a second set of neural networks 200B to both extractfeatures from Doppler modality images and to use the extracted featuresto classify the Doppler modality images by region (the neural networksused to extract features may be different than the neural network usedto classify the images). The first neural network 200A and the secondset of neural networks 200B may be implemented using convolutionalneural network (CNN) and may be referred to as classification neuralnetworks or CNNs.

Additionally, a third set of neural networks 200C, including adversarialnetworks, are employed for each classified 2D view type in order tosegment the cardiac chambers in the 2D images and produce segmented 2Dimages. A fourth set of neural networks 200D are used for eachclassified Doppler modality region in order to segment the Dopplermodality images to generate waveform traces. In additional embodiments,the machine learning layer 200 may further include a set of one or moreprediction CNNs for disease prediction and optionally a set of one ormore prognosis CNNs for disease prognosis (not shown). The third andfourth sets of neural networks 200C and 200D may be implemented usingCNNs and may be referred to as segmentation neural networks or CNNs.

In machine learning, a CNN is a class of deep, feed-forward artificialneural network typically use for analyzing visual imagery. Each CNNcomprises an input and an output layer, as well as multiple hiddenlayers. In neural networks, each node or neuron receives an input fromsome number of locations in the previous layer. Each neuron computes anoutput value by applying some function to the input values coming fromthe previous layer. The function that is applied to the input values isspecified by a vector of weights and a bias (typically real numbers).Learning in a neural network progresses by making incrementaladjustments to the biases and weights. The vector of weights and thebias are called a filter and represents some feature of the input (e.g.,a particular shape).

The machine learning layer 200 operates in a training mode to train eachof the CNNs 200A-200D prior to the echo workflow engine 12 being placedin an analysis mode to automatically recognize and analyze echo imagesin patient studies. In one embodiment, the CNNs 200A-200D may be trainedto recognize and segment the various echo image views using thousands ofecho images from an online public or private echocardiogram DICOMdatabase.

The presentation layer 202 is used to format and present information toa user. In one embodiment, the presentation layer is written in HTML 5,Angular 4 and/or JavaScript. The presentation layer 202 may include aWindows Presentation Foundation (WPF) graphical subsystem 202A forimplementing a light weight browser-based user interface that displaysreports and allows a user (e.g., doctor/technician) to edit the reports.The presentation layer 202 may also include an image viewer 202B (e.g.,a DICOM viewer) for viewing echo images, and a python server 202C forrunning the CNN algorithms and generating a file of the results inJavaScript Object Notation (JSON) format, for example.

The database layer 204 in one embodiment comprises a SQL database 204Aand other external services that the system may use. The SQL database204A stores patient study information for individual patient studiesinput to the system. In some embodiments, the database layer 204 mayalso include the image file archive 18 of FIG. 1.

FIG. 3 is a flow diagram illustrating one embodiment of a process forperformed by the echo workflow engine 12 to automatically recognize andanalyze both 2D and Doppler modality echo images to perform automatedmeasurements and the diagnosis, prediction and prognosis of heartdisease. The process occurs once the echo workflow engine 12 is trainedand placed in analysis mode.

The process may begin by the echo workflow engine 12 receiving from amemory one or more patient studies comprising a plurality ofechocardiogram images taken by an ultrasound device of a patient organ,such as a heart (block 300). In one embodiment, the patient study mayinclude 70-90 images and videos.

A first module of the echo workflow engine 12 may be used to operate asa filter to separate the plurality of echocardiogram images according to2D images and Doppler modality images based on analyzing image metadata(block 302). The first module analyzes the DICOM tags, or metadata,incorporated in the image, and runs an algorithm based upon the taginformation to distinguish between 2D and modality images, and thenseparate the modality images into either pulse wave, continuous wave,PWTDI or m-mode groupings. A second module of the echo workflow engine12 may perform color flow analysis on extracted pixel data using acombination of analyzing both DICOM tags/metadata and color contentwithin the images, to separate views that contain color from those thatdo not. A third module then anonymizes the data by removing metatagsthat contain personal information and cropping the images to exclude anyidentifying information. A fourth module then extracts the pixel datafrom the images and converts the pixel data to numpy arrays for furtherprocessing.

Because sonographers do not label the view types in the echo images, oneor more of neural networks are used classify the echo images by viewtype. In one embodiment, a first neural network is used by the echoworkflow engine 12 to classify the 2D images by view type (block 304);and a second set of neural networks is used by the echo workflow engine12 to extract the features from Doppler modality images and to use theextracted features to classify the Doppler modality images by region(block 306). As shown, the processing of 2D images is separate from theprocessing of Doppler modality images. In one embodiment, the firstneural network and the second set of neural networks may implementedusing the set of classification convolutional neural networks (CNNs)200A. In one specific embodiment, a five class CNN may be used toclassify the 2D images by view type and an 11 class CNN may be used toclassify the Doppler modality images by region. Each of the neuralnetworks, and classifying by view type, can use a majority voting schemeto determine the optimal answer. For example a video can be divided intostill image frames, and each frame may be given a classification label,i.e., of a vote, and the classification label receiving the most votesis applied to classify the video.

In one embodiment, the echo workflow engine 12 is trained to classifymany different view types. For example, the echo workflow engine 12 mayclassify at least 11 different view types including parasternal longaxis (FLAX), apical 2-, 3-, and 4-chamber (A2C, A3C, and A4C), A4C pluspulse wave of the mitral valve, A4C plus pulse wave tissue Doppler onthe septal side, A4C plus pulse wave tissue Doppler on the lateral side,A4C plus pulse wave tissue Doppler on the tricuspid side, A5C pluscontinuous wave of the aortic valve, A4C+Mmode (TrV), A5C+PW (LVOT).

Based on the classified images, a third set of neural networks is usedby the echo workflow engine 12 to segment regions of interest (e.g.,cardiac chambers) in the 2D images to produce annotated or segmented 2Dimages (block 308). A fourth set of neural networks is used by the echoworkflow engine 12 for each classified Doppler modality region togenerate waveform traces and to generate annotated or segmented Dopplermodality images (block 309). The process of segmentation includesdetermining locations where each of the cardiac chambers begin and endto generate outlines of structures of the heart (e.g., cardiac chambers)depicted in each image and/or video. Segmentation can also be used totrace the outline of the waveform depicting the velocity of blood flowin a Doppler modality. In one embodiment, the third and fourth sets ofneural networks maybe referred to as segmentation neural networks and mycomprise the set of segmentation CNNs 200B and 200C. The choice ofsegmentation CNN used is determined by the view type of the image, whichmakes the prior correct classification of view type a crucial step. In afurther embodiment, once regions of interest are segmented, a separateneural network can be used to smooth outlines of the segmentations.

As will be explained further below, the segmentation CNNs may be trainedfrom hand-labeled real images or artificial images generated by generaladversarial networks (GANs).

Using both the segmented 2D images and the segmented Doppler modalityimages, the echo workflow engine 12 calculates for the patient study,measurements of cardiac features for both left and right sides of theheart (block 310).

The echo workflow engine 12 then generates conclusions by comparing thecalculated measurements with international cardiac guidelines (block312). The echo workflow engine 12 further outputs at least one report toa user highlighting ones of the calculated measurements that falloutside of the International guidelines (block 314). In one embodiment,two reports are generated and output: the first report is a list of thecalculated values for each measurement with the highest confidence asdetermined by a rules based engine, highlighting values among thecalculated measurements that fall outside of the Internationalguidelines; and the second report is a comprehensive list of allmeasurements calculated on every image frame of every video, in everyview, generating large volumes of data. All report data and extractedpixel data may be stored in a structured database to enable machinelearning and predictive analytics on images that previously lacked thequantification and labelling necessary for such analysis. The structureddatabase may be exported to a cloud based server or may remain onpremises (e.g., of the lab owning the images) and can be connected toremotely. By connecting these data sources into a single network,disease prediction algorithms can be progressively trained acrossmultiple network nodes, and validated in distinct patient cohorts. Inone embodiment, the reports may be electronically displayed to a doctorand/or a patient on a display of an electronic device and/or as a paperreport. In some embodiments, the electronic reports may be editable bythe user per rule or role based permissions, e.g., a cardiologist may beallowed to modify the report, but a patient may have only viewprivileges.

FIG. 4A is a flow diagram illustrating further details of the processfor automatically recognizing and analyze both 2D and Doppler modalityecho images to perform automated measurements and the diagnosis,prediction and prognosis of heart disease according to one embodiment.

The process may begin with receiving one or more patient studies (FIG. 3block 300), which comprises blocks 400-4010. In one embodiment, echoimages from each of the patient studies are automatically downloadedinto the image file archive 18 (block 400). The echo images may bereceived from a local or remote storage source of the computer 14. Thelocal storage sources may include internal/external storage of thecomputer 14 including removable storage devices. The remote storagesources may include the ultrasound imaging device 24, the DICOM servers30, the network file share devices 32, the echo workstations 34, and/orthe cloud storage services 36 (see FIG. 1). In one embodiment, the echoworkflow engine 12 includes functions for operating as a picturearchiving and communication server (PACS), which is capable of handlingimages from multiple modalities (source machine types, one of which isthe ultrasound imaging device 24). The echo workflow engine 12 uses PACSto download and store the echo images into the image file archive 18 andprovides the echo workflow engine 12 with access to the echo imagesduring the automated workflow. The format for PACS image storage andtransfer is DICOM (Digital Imaging and Communications in Medicine).

Patient information from each of the patient studies is extracted andstored in the database 16 (block 402). Non-image patient data mayinclude metadata embedded within the DICOM images and/or scanneddocuments, which may be incorporated using consumer industry standardformats such as PDF (Portable Document Format), once encapsulated inDICOM. In one embodiment, received patient studies are placed in aprocessing queue for future processing, and during the processing ofeach patient study, the echo workflow engine 12 queues and checks forunprocessed echo images (block 404). The echo workflow engine 12monitors the status of patient studies, and keeps track of them in aqueue to determine which have been processed and which are stillpending. In one embodiment, prioritization of the patient studies in thequeue may be configured by a user. For example, the patient studies maybe prioritized in the queue for processing according to the date of theecho exam, the time of receipt of the patient study or by estimatedseverity of the patient's heart disease.

Any unprocessed echo images are then filtered for having a valid DICOMimage format and non DICOM files in an echo study are discarded (block406). In one embodiment, the echo images are filtered for having aparticular type of format, for example, a valid DICOM file format, andany other file formats may be ignored. Filtering the echo images forhaving a valid image file format enhances the reliability of the echoworkflow engine 12 by rejecting invalid DICOM images for processing.

Any unprocessed valid echo images are then opened and processed in thememory of the computer 14 (block 408). Opening of the echo images forthe patient study in memory of the computer 14 is done to enhanceprocessing speed by echo workflow engine 12. This is in contrast to anapproach of opening the echo files as sub-processes, saving the echofiles to disk, and then reopening each echo image during processing,which could significantly slow processing speed.

The echo workflow engine 12 then extracts and stores the metadata fromthe echo images and then anonymizes the images by blacking out theimages and overwriting the metadata in order to protect patient dataprivacy by covering personal information written on the image (block410). As an example, DICOM formatted image files include metadatareferred to as DICOM tags that may be used to store a wide variety ofinformation such as patient information, Doctor information, ultrasoundmanufacture information, study information, and so on. In oneembodiment, the extracted metadata may be stored in the database 16 andthe metadata in image files is over written for privacy.

After receipt and processing of the patient studies, the echo workflowengine 12 separates 2D images from Doppler modality images so the twodifferent image types can be processed by different pipeline flows,described below. In one embodiment, the separating of the images (FIG. 3block 302) may comprise blocks 412-414. First, the 2D images areseparated from the Doppler modality images by analyzing the metadata(block 412).

FIGS. 5A and 5B are diagrams illustrating an example 2D echo image 500and an example Doppler modality image 502 including a waveform,respectively. The echo workflow engine 12 may determine the image typeby examining metadata/DICOM tags. In one embodiment, information withinthe DICOM tags may be extracted in order to group the images into one ofthe following four classes: 2D only, pulsed-wave (PW), continuous wave(CW), and m-mode. Similarly, the transducer frequency of the ultrasoundimaging device 24 in the metadata may be used to further separate someof the PW images into a fifth class: pulsed-wave tissue doppler imaging(PWTDI).

Referring again to FIG. 4A, the echo workflow engine 12 may also filterout images with a zoomed view, which may also be determined by analyzingthe metadata (block 414). Any of the echo images that has been zoomedduring image capture are not processed through the pipeline because whenzooming, useful information is necessarily left out of the image,meaning the original image would have to be referenced for the missingdata, which is a duplication of effort that slows processing speed.Accordingly, rather than potentially slowing the process in such amanner, the echo workflow engine 12 filters out or discards the zoomedimages to save processing time. In an alternative embodiment, filteringout zoomed images in block 414 may be performed prior to separating theimages in block 412.

After separating the 2D images from the Doppler modality images, theecho workflow engine 12 extracts and converts the image data from eachecho image into numerical arrays 416 (block 416). For the echo imagesthat are 2D only, the pixel data comprises a series of image framesplayed in sequence to create a video. Because the image frames areunlabeled, the view angle needs to be determined. For the Dopplermodality images that include waveform modalities, there are two imagesin the DICOM file that may be used for subsequent view identification, awaveform image and an echo image of the heart. The pixel data isextracted from the DICOM file and tags in the DICOM file determine thecoordinates to crop the images. The cropped pixel data is stored innumerical arrays for further processing. In one embodiment, blocks 412,414 and 416 may correspond to the separating images block 302 of FIG. 3.

After separating images, the echo workflow engine 12 attempts toclassify each of the echo images by view type. In one embodiment, viewclassification (FIG. 3 blocks 304 and 306) correspond to blocks 418-422.

According to the disclosed embodiments, the echo workflow engine 12attempts to classify each of the echo images by view type by utilizingparallel pipeline flows. The parallel pipeline includes a 2D imagepipeline and a Doppler modality image pipeline. The 2D pipeline flowbegins by classifying, by a first CNN, the 2D images by view type (block418), corresponding to block 304 from FIG. 3. The Doppler modality imagepipeline flow begins by classifying, by a second CNN, the Dopplermodality images by view type (block 420), corresponding to block 306from FIG. 3.

FIGS. 6A-6K are diagrams illustrating some example view typesautomatically classified by the echo workflow engine 12. As statedpreviously, example view types may include parasternal long axis (FLAX),apical 2-, 3-, and 4-chamber (A2C, A3C, and A4C), A4C plus pulse wave ofthe mitral valve, A4C plus pulse wave tissue Doppler on the septal side,A4C plus pulse wave tissue Doppler on the lateral side, A4C plus pulsewave tissue Doppler on the tricuspid side, A5C plus continuous wave ofthe aortic valve, A4C+Mmode (TrV), A5C+PW (LVOT).

Referring again to FIG. 4A, in one embodiment, 2D image classificationis performed as follows. If the DICOM file contains video frames from a2D view, only a small subset of the video frames are analyzed todetermine 2D view classification for more efficient processing. In oneembodiment, the subset of the video frames may range approximately 8-12video frames, but preferably 10 frames are input into one of the CNNs200A trained for 2D to determine the actual view. In an alternativeembodiment, subset a video frames may be randomly selected from thevideo file. In one embodiment, the CNNs 200A classify each of theanalyzed video frames as one of: A2C, A3C, A4C, ASC, PLAX Modified,PLAX, PSAX AoV level, PSAX Mid-level, Subcostal Ao, Subcostal Hep vein,Subcostal IVC, Subcostal LAX, Subcostal SAX, Suprasternal and Other.

Doppler modality images comprise two images, an echocardiogram image ofthe heart and a corresponding waveform, both of which are extracted fromthe echo file for image processing. In one embodiment, Doppler modalityimage classification of continuous wave (CW), pulsed-wave (PW), andM-mode images is performed as follows. If the DICOM file contains awaveform modality (CW, PW, PWTDI, M-mode), the two extracted images areinput to one of the CNNs 200A trained for CW, PW, PWTDI and M-mode viewclassification to further classify the echo images as one of: CW (AoV),CW (TrV), CW Other, PW (LVOT), PW (MV), PW Other, PWTDI (lateral), PWTDI(septal), PWTDI (tricuspid), M-mode (TrV) and M-mode Other.

There are many more potential classifications available for modalities,but the present embodiments strategically select the classes above,while grouping the remaining potential classes into “Other”, in order tomaximize processing efficiency, while identifying the most clinicallyimportant images for further processing and quantification.Customization of the CNNs 200A occurs in the desired number of layersused and the quantity of filters within each layer. During the trainingphase, the correct size of the CNNs may be determined through repeatedtraining and adjustments until optimal performance levels are reached.

During view classification, the echo workflow engine 12 maintainsclassification confidence scores that indicate a confidence level thatthe view classifications are correct. The echo workflow engine 12filters out the echo images having classification confidence scores thatfail to meet a threshold, i.e., low classification confidence scores(block 422). Multiple algorithms may be employed to deriveclassification confidence scores depending upon the view in question.Anomalies detected in cardiac structure annotations, image quality,cardiac cycles detected and the presence of image artifacts may allserve to decrease the classification confidence score and discard anecho image out of the automated echo workflow.

With respect to the confidence scores, the echo workflow engine 12generates and analyzes several different types of confidence scores atdifferent stages of processing, including classification, annotation,and measurements (e.g., blocks 422, 434 and 442). For example, poorquality annotations or classifications, which may be due to substandardimage quality, are filtered out by filtering the classificationconfidence scores. In another example, in a patient study the same viewmay be acquired more than once, in which case the best measurements arechosen by filtering out low measurement confidence scores as describedfurther below in block 442. Any data having a confidence score thatmeets a predetermined threshold continues through the workflow. Shouldthere be a duplication of measurements both with high confidence, themost clinically relevant measurement may be chosen.

Next, the echo workflow engine 12 performs image segmentation to defineregions of interest (ROI). In computer vision, image segmentation is theprocess of partitioning a digital image into multiple segments (sets ofpixels) to locate and boundaries (lines, curves, and the like) ofobjects. Typically, annotations are a series of boundary linesoverlaying overlaid on the image to highlight segment boundaries/edges.In one embodiment, the segmentation to define ROI (FIG. 3 block 308)corresponds to blocks 426-436.

In one embodiment, the 2D image pipeline annotates, by a third CNN,regions of interests, such as cardiac chambers in the 2D images, toproduce annotated 2D images (block 426). An annotation post process thenerodes the annotations to reduce their dimensions, spline fits outlinesof cardiac structures and adjusts locations of the boundary lines closerto the region of interest (ROIs) (block 427). The 2D image pipelinecontinues with analyzing the ROIs (e.g., cardiac chambers) in theannotated 2D images to estimate volumes and determine key points in thecardiac cycle by finding systolic/diastolic end points (block 430). For2D only views, measurements are taken at the systolic or diastolic phaseof the cardiac cycle, i.e. when the left ventricle reaches the smallestvolume (systole) or the largest volume (diastole). From the 2D videoimages, it must be determined which end points are systolic and whichare diastolic based on the size of the estimated volumes of the leftventricle. For example, a significantly large left ventricle mayindicate a dystonic end point, while a significantly small volume mayindicate a systolic end point. Every video frame is annotated and thevolume of the left ventricle is calculated throughout the whole cardiaccycle. The frames with minimum and maximum volumes are detected with apeak detection algorithm.

The Doppler modality pipeline analyzes the Doppler modality images andgenerates, by a fourth CNN, a mask and a waveform trace in the Dopplermodality images to produce annotated Doppler modality images (block431).

FIG. 7 is a diagram illustrating an example 2D image segmented in block426 to produce annotations 700 indicating cardiac chambers, and anexample Doppler modality image segmented in block 431 to produce a maskand waveform trace 702.

In one embodiment, the third and fourth CNNs may correspond tosegmentation CNNs 200B. In one embodiment, each of the CNNs 200B used tosegment the 2D images and Doppler modality images may be implemented asU-Net CNN, which is convolutional neural network developed forbiomedical image segmentation. Multiple U-Nets may be used. For example,for 2D images, a first U-Net CNN can be trained to annotate ventriclesand atria of the heart from the A2C, A3C, A4C views. A second U-net CNNcan be trained to annotate the chambers in the PLAX views. For M-modeviews, a third U-Net CNN can be trained to segment the waveform, removesmall pieces of the segments to find likely candidates for the region ofinterest, and then reconnect the segments to provide a full trace of themovement of the mitral valve. For CW views, a fourth U-net CNN can betrained to annotate and trace blood flow. For PW views, a fifth U-netCNN trained to annotate and trace the blood flow. For PWTDI views, asixth U-net CNN can be trained to annotate and trace movement of thetissues structures (lateral/septal/tricuspid valve).

Referring again to FIG. 4A, the Doppler modality pipeline continues byprocessing the annotated Doppler modality images with a sliding windowto identify cycles, peaks are measured in the cycles, and key points inthe cardiac cycle are determined by finding systolic/diastolic endpoints (block 432). Typically a Doppler modality video may capture threeheart cycles and the sliding window is adjusted in size to block out twoof the cycles so that only one selected cycle is analyzed. Within theselected cycle, the sliding window is used to identify cycles, peaks aremeasured in the cycles, and key points in the cardiac cycle aredetermined by finding systolic/diastolic end points.

FIGS. 8A and 8B are diagrams illustrating examples of findingsystolic/diastolic end points in the cardiac cycle, for both 2D andDoppler modalities, respectively, which are key points in order to takeaccurate cardiac measurements

Referring again to FIG. 4A, in one embodiment, the echo workflow engine12 maintains annotation confidence scores corresponding to the estimatedvolumes, systolic/diastolic end points, identified cycles and measuredpeaks. The echo workflow engine 12 filters out annotated images havingannotation confidence scores that fail to meet a threshold, i.e., lowannotated confidence scores (block 434). Examples of low confidenceannotations may include annotated images having one or more of:excessively small areas/volumes, sudden width changes, out of proportionsectors, partial heart cycles, and insufficient heart cycles.

After images having low annotation confidence scores are filtered out,the echo workflow engine 12 defines an imaging window for each image,and filters out annotations that lie outside of the imaging window(block 435).

FIGS. 9A and 9B are diagrams illustrating processing of an imagingwindow in a 2D echo image 900. In FIG. 9A, a shaded ROI 902 is shownhaving unshaded regions or holes 904 therein. Open CV morphologytransformations are used to fill the holes 904 inside the ROI, as shownin In FIG. 9B. Thereafter, a Hough line transformation may be used tofind an imaging window border 906. As is well known, a Hough transformis a feature extraction technique used in digital image processing tofind imperfect instances of objects within a certain class of shapes.After an imaging window border is are found, a pixel account ofannotations beyond the imaging window border is made. Annotations 908with a significant number of pixels outside the border of the imagingwindow are then discarded.

Referring again to FIG. 4A, in the handheld configuration for theautomated clinical workflow system 10C (See FIG. 1B), the patientstudies may not include Doppler modality images. According to a furtheraspect, the disclosed embodiments accommodate for such handheldconfigurations by using the 2D images to simulate Doppler modalitymeasurements by using Left Ventricular (LV) and Left Atrial (LA) volumemeasurements to derive E, e′ and A (e.g., early and late diastolictransmittal flow and early/mean diastolic tissue velocity) measurements(block 436). In one embodiment, simulating the Doppler modalitymeasurements may be optional and may be invoked based on a softwaresetting indicating the presence of a handheld configuration and/orabsence of Doppler modality images for the current patient study.

Referring again to FIG. 4A, in one embodiment, once cardiac features areannotated during segmentation, the cardiac features are then measuredduring a measurement process (block 310 of FIG. 3), which in oneembodiment may comprises block 438-448. The process of measuring cardiacfeatures may begin by quantifying a plurality of measurements using theannotations. First, the 2D pipeline measures for the 2D imagesleft/right ventricle, left/right atriums, left ventricular outflow(LVOT) and pericardium (block 438). For the Doppler modality images, theDoppler modality image pipeline measures blood flow velocities (block440).

More specifically, for A2C, A3C, A4C, and A5C image views, volumetricmeasurements of chamber size are conducted on the systolic and diastolicframes of the video, and image processing techniques mimic a trainedclinician at measuring the volume using the method of disks (MOD). For2D Plax, PSAX (mid level), PSAX (AoV level), Subcostal, Suprasternal andIVC image views, linear measurements of chamber size and inter-chamberdistances are conducted on the systolic and diastolic frames of thevideo using image processing techniques to mimic the trained clinician.For M-mode image views, from the annotated segments of the movement ofthe Tricuspidvalve, a center line is extracted and smoothed, and thenthe peaks and valleys are measured in order to determine the minimum andmaximum deviations over the cardiac cycle. For PW image views, from theannotations of the blood flow, a mask is created to isolate parts of thewaveform. A sliding window is then run across the trace to identify onefull heart cycle, in combination with heart rate data from the DICOMtags, to use as a template. This template is then used to identify allother heart cycles in the image. Peak detection is then performed oneach cycle and then run through an algorithm to identify which part ofthe heart cycle each peak represents. For CW image views, from theannotations of the trace of the blood flow, curve fitting is performedon the annotation to then quantify the desired measurements. For PWTDIimage views, from the annotations of the movement of the tissue, a maskis created to isolate parts of the waveform. A sliding window is thenrun across the trace to identify one full heart cycle, in combinationwith heart rate data from the DICOM tags, to use as a template. Thistemplate is then used to identify all other heart cycles in the image.Peak detection is then performed on each cycle and then run through analgorithm to identify which part of the heart cycle each peakrepresents.

FIGS. 10A and 10B are diagrams graphically illustrating structuralmeasurements automatically generated from annotations of cardiacchambers in a 2D image, and velocity measurements automaticallygenerated from annotations of waveforms in a Doppler modality,respectively.

The measurement table below list the measurements that may be compiledby the echo workflow engine 12 according to one embodiment.

Measurement Table

Measurement Name Measurement Description LAESV MOD A2C Left Atrial EndSystolic Volume in A2C calculation based on Method Of Discs LAESVi MODA2C Left Atrial End Systolic Volume in A2C calculation based on MethodOf Discs indexed to BSA LAL A2C Left Atrial End Sysolic Length measuredin A2C LVEDV MOD A2C Left Ventricular End Diastolic Volume in A2Ccalculation based on Method Of Discs LVEDVi MOD A2C Left Ventricular EndDiastolic Volume in A2C calculation based on Method Of Discs indexed toBSA LVEF MOD A2C Left Ventricular Ejection Fraction in A2C based onMethod Of Discs LVESV MOD A2C Left Ventricular End Systolic Volume inA2C calculation based on Method Of Discs LVESVi MOD A2C Left VentricularEnd Systolic Volume in A2C calculation based on Method Of Discs indexedto BSA LV length A2C Left Ventricular Length measured in A2C LAESV MODA4C Left Atrial End Systolic Volume in A4C calculation based on MethodOf Discs LAESVi MOD A4C Left Atrial End Systolic Volume in A4Ccalculation based on Method Of Discs indexed to BSA LAL A4C Left AtrialEnd Systolic Length measured in A4C LAW A4C Left Atrial End SystolicWidth measurement in A4C LA area A4C Left Atrial Area measured in A4CLAESV A-L A4C Left Atrial End Systolic Volume in A4C calculation basedon Area-Length method LVEDV MOD A4C Left Ventricular End DiastolicVolume in A4C calculation based on Method Of Discs LVEDVi MOD A4C LeftVentricular End Diastolic Volume in A4C calculation based on Method OfDiscs indexed to BSA LVEF MOD A4C Left Ventricular Ejection Fraction inA4C based on Method Of Discs LVESV MOD A4C Left Ventricular End SystolicVolume in A4C calculation based on Method Of Discs LVESVi MOD A4C LeftVentricular End Systolic Volume in A4C calculation based on Method OfDiscs indexed to BSA LV length A4C Left Ventricular Length measured inA4C LVAd A4C Left Ventricular Area measured at end diastole in A4C TAPSETricuspid Annular Plane Systolic Excursion DecT Deceleration Time ofearly diastolic MV transmitral flow E/A ratio Ratio of early and latediastolic transmitral flow MV-A Late diastolic transmitral flow MV-AdurDuration of late diastolic transmitral flow MV-E Early diastolictransmitral flow e′ lateral Early diastolic tissue velocity taken fromthe lateral region e′ mean Mean early diastolic tissue velocity (mean oflateral and septal region) e′ septal Early diastolic tissue velocitytaken from the septal region E/e′ lateral Ratio of early transmitralflow and early diastolic tissue velocity taken form the lateral regionE/e′ septal Ratio of early transmitral flow and early diastolic tissuevelocity taken form the septal region E/e′ mean Ratio of earlytransmitral flow and mean diastolic tissue velocity a′ lateral Latediastolic tissue velocity taken from the lateral region a′ septal Latediastolic tissue velocity taken from the septal region s′ lateralSystolic tissue velocity taken from the lateral region s′ septalSystolic tissue velocity taken from the septal region LAESV A-L biplaneLeft Atrial End Systolic Volume biplane calculation based on Area-Lengthmethod LAESV MOD biplane Left Atrial End Systolic Volume biplanecalculation based on Method Of Discs LAESVi MOD biplane Left Atrial EndSystolic Volume biplane calculation based on Method Of Discs indexed toBSA LAESVi A-L biplane Left Atrial End Systolic Volume biplanecalculation based on Area-Length method indexed to BSA LVCO MOD biplaneLeft Ventricular Cardiac Output biplane calculation based on Method OfDiscs LVEDV MOD biplane Left Ventricular End Diastolic Volume biplanecalculation based on Method Of Discs LVEDVi MOD biplane Left VentricularEnd Diastolic Volume biplane calculation based on Method Of Discsindexed to BSA LVEF MOD biplane Left Ventricular Ejection Fractionbiplane based on Method Of Discs LVESV MOD biplane Left Ventricular EndSystolic Volume biplane calculation based on Method Of Discs LVESVi MODbiplane Left Ventricular End Systolic Volume biplane calculation basedon Method Of Discs indexed to BSA LVSV MOD biplane Left VentricularStroke Volume biplane calculation based on Method of Disks AoV VmaxAortic Valve maximum Velocity AoV Vmean Aortic Valve mean Velocity AoVPmax Aortic Valve maximum Pressure gradient LVOT Vmax Left VentricularOutflow Tract maximum Velocity LVOT Vmean Left Ventricular Outflow Tractmean Velocity IVSd Inter Ventricular Septal thickness measured enddiastolic LV mass Left Ventricular mass LVIDd Left Ventricular internalDiameter measured at end diastole LVIDd index Left Ventricular internalDiameter measured at end diastole indexed to BSA LVIDs Left Ventricularinternal Diameter measured at end systole LVIDs index Left Ventricularinternal Diameter measured at end systole indexed to BSA LVMi LeftVentricular Mass indexed to BSA LVOT Left Ventricular Outflow Tractdiameter LVPWd Left Ventricular Posterior Wall thickness measured enddiastolic RWT Relative Wall Thickness LA area A2C Left Atrial Areameasured in A2C LAESV A-L A2C Left Atrial End Systolic Volume in A2Ccalculation based on Area-Length method LVAd A2C Left Ventricular Areameasured at end diastole in A2C LVAs A2C Left Ventricular Area measuredat end systole in A2C LVAs A4C Left Ventricular Area measured at endsystole in A4C PASP Pulmonary Artery Systolic Pressure RAL Right AtrialEnd Systolic Length RAW Right Atrial End Systolic Width RAESV MOD A4CRight Atrial end systolic Volume in A4C calculation based on Method OfDiscs RAESV A-L A4C Right Atrial end systolic Volume in A4C calculationbased on Area-Length method RAESVi MOD A4C Right Atrial end systolicVolume in A4C calculation based on Method Of Discs indexed to BSA RAarea Right Atrial area RVIDd Right Ventricular End Diastolic InternalDiameter RV area (d) Right Ventricular Area (measured at end-diastole)RV area (s) Right Ventricular Area (measured at end systole) LVOT PmaxLeft Ventricular Outflow Tract max pressure gradient LVOT Pmean LeftVentricular Outflow Tract mean pressure gradient LVSV (Doppler) LeftVentricular Stroke Volume based on Doppler LVOT VTI Left VentricularOutflow Tract Velocity Time Integral LVCO (Doppler) Left VentricularCardiac Output (based on Doppler) LVCOi (Doppler) Left VentricularCardiac Output (based on Doppler) indexed to Body Surface Area LVSVi(Doppler) Left Ventricular Stroke Volume (based on Doppler) indexed toBody Surface Area TR Vmax Tricuspid Regurgitation maximum velocity CSALVOT Crossectional Area of the LVOT Sinotub J Sinotubular junctiondiameter Sinus valsalva Sinus valsalva diameter Asc. Ao Ascending Aortadiameter Asc. Ao index Ascending Aorta diameter index Sinus valsalvaindex Sinus valsalva diameter indexed to BSA IVC max Inferior Vena Cavamaximum diameter IVC min Inferior Vena Cava minimum diameter IVC CollapsInferior Vena Cava collaps RVIDd mid Right Ventricular Internal Diameterat mid level (measured end diastole) RVOT prox Right Ventricular OutflowTract proximal diameter RVOT dist Right Ventricular Outflow Tract distaldiameter RV FAC Right Ventricular Fractional Area Change TE I indexRVAWT Right Ventricular Anterior Wall Thickness TrV-E Tricuspid valve Ewave TrV-A Tricuspid valve A wave TrV E/A Tricuspid valve E/A ratio TrVDecT Tricuspid valve deceleration time MV Vmax Mitral valve maximumvelocity MV Vmean Mitral valve mean velocity MV VTI Mitral valvevelocity time intergal MV PHT Mitral valve pressure half time MVA (byPHT) Mitral valve area (by pressure half time) RV e′ Early diastolictissue velocity taken from the right ventricular free wall region RV a′Late diastolic tissue velocity taken from the right ventricular freewall region RV s′ Systolic tissue velocity taken from the rightventricular free wall region RVCO Right Ventricular Cardiac Output ULSUnidimensional Longitudinal Strain Ao-arch Aortic arch diameterDescending Ao Descending Aortic diameter Ao-arch index Aortic archdiameter indexed to BSA Descending Ao index Descending Aortic diameterindexed to BSA LA GLS (reservoir) (A4C) Left Atrial strain duringsystole measured in A4C LA GLS (conduit) (A4C) Left Atrial strain duringearly diastole measured in A4C LA GLS (booster) (A4C) Left Atrial strainduring pre atrial contraction measured in A4C LA GLS (reservoir) (A2C)Left Atrial strain during systole measured in A2C LA GLS (conduit) (A2C)Left Atrial strain during early diastole measured in A2C LA GLS(booster) (A2C) Left Atrial strain during pre atrial contractionmeasured in A2C LA GLS (reservoir) Left Atrial strain during systole LAGLS (conduit) Left Atrial strain during early diastole LA GLS (booster)Left Atrial strain during pre atrial contraction LVSr-e (A4C) LeftVentricular strain rate during early diastole measured in A4C LVSr-a(A4C) Left Ventricular strain rate during late diastole measured in A4CLVSr-s (A4C) Left Ventricular strain rate during systole measured in A4CLVSr-e (A2C) Left Ventricular strain rate during early diastole measuredin A2C LVSr-a (A2C) Left Ventricular strain rate during late diastolemeasured in A2C LVSr-s (A2C) Left Ventricular strain rate during systolemeasured in A2C LVSr-e Left Ventricular strain rate during earlydiastole LVSr-a Left Ventricular strain rate during late diastole LVSr-sLeft Ventricular strain rate during systole LASr-e (A4C) Left Atrialstrain rate during early diastole LASr-a (A4C) Left Atrial strain rateduring late diastole LASr-s (A4C) Left Atrial strain rate during systoleLASr-e (A2C) Left Atrial strain rate during early diastole LASr-a (A2C)Left Atrial strain rate during late diastole LASr-s (A2C) Left Atrialstrain rate during systole LASr-e Left Atrial strain rate during earlydiastole LASr-a Left Atrial strain rate during late diastole LASr-s LeftAtrial strain rate during systole AV-S (A4C) Atrio Ventricular strainmeasured in A4C AV-S (A2C) Atrio Ventricular strain measured in A2C AV-SAtrio Ventricular strain Sr-Sav (A4C) Atrio Ventricular strain rateduring systole measured in A4C Sr-Eav (A4C) Atrio Ventricular strainrate during early diastole measured in A4C Sr-Aav (A4C) AtrioVentricular strain rate during late diastole measured in A4C Sr-Sav(A2C) Atrio Ventricular strain rate during systole measured in A2CSr-Eav(A2C) Atrio Ventricular strain rate during early diastole measuredin A2C Sr-Aav (A2C) Atrio Ventricular strain rate during late diastolemeasured in A2C Sr-Sav Atrio Ventricular strain rate during systoleSr-Eav Atrio Ventricular strain rate during early diastole Sr-Aav AtrioVentricular strain rate during late diastole LVVr-e (A4C) LeftVentricular volume rate during early diastole measured in A4C LVVr-a(A4C) Left Ventricular volume rate during late diastole measured in A4CLVVr-s (A4C) Left Ventricular volumerate during systole measured in A4CLVVr-e (A2C) Left Ventricular volume rate during early diastole measuredin A2C LVVr-a (A2C) Left Ventricular volume rate during late diastolemeasured in A2C LVVr-s (A2C) Left Ventricular volumerate during systolemeasured in A2C LVVr-e Left Ventricular volume rate during earlydiastole LVVr-a Left Ventricular volume rate during late diastole LVVr-sLeft Ventricular volumerate during systole LAVr-e (A4C) Left Atrialvolume rate during early diastole measured in A4C LAVr-a (A4C) LeftAtrial volume rate during late diastole measured in A4C LAVr-s (A4C)Left Atrial volumerate during systole measured in A4C LAVr-e (A2C) LeftAtrial volume rate during early diastole measured in A2C LAVr-a (A2C)Left Atrial volume rate during late diastole measured in A2C LAVr-s(A2C) Left Atrial volumerate during systole measured in A2C LAVr-e LeftAtrial volume rate during early diastole LAVr-a Left Atrial volume rateduring late diastole LAVr-s Left Atrial volumerate during systole TLVdTotal Left heart volume end-diastolic TLVs Total Left heart volumeend-systolic TLVd (A4C) Total Left heart volume end-diastolic measuredin A4C TLVs (A4C) Total Left heart volume end-systolic measured in A4CTLVd (A2C) Total Left heart volume end-diastolic measured in A2C TLVs(A2C) Total Left heart volume end-systolic measured in A2C Ar Pulmonaryvein Atrial reversal flow Ardur Pulmonary vein Atrial reversal flowduration D Pulmonary vein diastolic flow velocity S Pulmonary veinsystolic flow velocity S/D ratio Ratio of Pulmonary vein systolic- anddiastolic flow Vel. RV GLS Right Ventricular Global Longitudinal Strain(mean) RV GLS (A4C) Right Ventricular Global Longitudinal Strainmeasured in A4C RV GLS (A2C) Right Ventricular Global LongitudinalStrain measured in A2C RV GLS (A3C) Right Ventricular GlobalLongitudinal Strain measured in A3C LA GLS Left Atrial GlobalLongitudinal Strain (mean) LA GLS (A4C) Left Atrial Global LongitudinalStrain measured in A4C LA GLS (A2C) Left Atrial Global LongitudinalStrain measured in A2C LA GLS (A3C) Left Atrial Global LongitudinalStrain measured in A3C RA GLS Right Atrial Global Longitudinal Strain(mean) RA GLS (A4C) Right Atrial Global Longitudinal Strain measured inA4C RA GLS (A2C) Right Atrial Global Longitudinal Strain measured in A2CRA GLS (A3C) Right Atrial Global Longitudinal Strain measured in A3C PVVmax Pulmonary Valve maximum Velocity PV Vmean Pulmonary Valve meanVelocity PV Pmax Pulmonary Valve maximum Pressure gradient PV PmeanPulmonary Valve mean Pressure gradient PV VTI Pulmonary Valve VelocityTime Integral MV-Adur-Ardur Difference between late diastolictransmitral flow and pulmonary vein atrial reversal flow duration APCArteria pulmonalis communis LV eccentricity index LV eccentricity indexMean % WT A2C Mean percentual Wall Thickening of 6 segments in A2C AA-%WT Percentile wall thickening of apical anterior segment AA-WTd Wallthickness of apical anterior segment in diastole AA-WTs Wall thicknessof apical anterior segment in systole Al-% WT Percentile wall thickeningof apical inferior segment Al-WTd Wall thickness of apical inferiorsegment in diastole Al-WTs Wall thickness of apical inferior segment insystole BA-% WT Percentile wall thickening of basal anterior segmentBA-WTd Wall thickness of basal anterior segment in diastole BA-WTs Wallthickness of basal anterior segment in systole BI-% WT Percentile wallthickening of basal interior segment BI-WTd Wall thickness of basalinterior segment in diastole BI-WTs Wall thickness of basal interiorsegment in systole MA-% WT Percentile wall thickening of mid anteriorsegment MA-WTd Wall thickness of mid anterior segment in diastole MA-WTsWall thickness of mid anterior segment in systole MI-% WT Percentilewall thickening of mid inferior segment MI-WTd Wall thickness of midinferior segment in diastole MI-WTs Wall thickness of mid inferiorsegment in systole Pericardial effusion Pericardial effusion Mean % WTA3C Mean percentual Wall Thickening of 6 segments in A3C AAS-% WTPercentile wall thickening of apical antero-septal segment AAS-WTd Wallthickness of apical antero-septal segment in diastole AAS-WTs Wallthickness of apical antero-septal segment in systole AP-% WT Percentilewall thickening of apical posterior segment AP-WTd Wall thickness ofapical posterior segment in diastole AP-WTs Wall thickness of apicalposterior segment in systole BAS-% WT Percentile wall thickening ofbasal antero-septal segment BAS-WTd Wall thickness of basalantero-septal segment in diastole BAS-WTs Wall thickness of basalantero-septal segment in systole BP-% WT Percentile wall thickening ofbasal posterior segment BP-WTd Wall thickness of basal posterior segmentin diastole BP-WTs Wall thickness of basal posterior segment in systoleMAS-% WT Percentile wall thickening of mid antero-septal segment MAS-WTdWall thickness of mid antero-septal segment in diastole MAS-WTs Wallthickness of mid antero-septal segment in systole MP-% WT Percentilewall thickening of mid posterior segment MP-WTd Wall thickness of midposterior segment in diastole MP-WTs Wall thickness of mid posteriorsegment in systole Mean % WT A4C Mean percentual Wall Thickening of 6segments in A4C AL-% WT Percentile wall thickening of apical lateralsegment AL-WTd Wall thickness of apical lateral segment in diastoleAL-WTs Wall thickness of apical lateral segment in systole AS-% WTPercentile wall thickening of apical septal segment AS-WTd Wallthickness of apical septal segment in diastole AS-WTs Wall thickness ofapical septal segment in systole BL-% WT Percentile wall thickening ofbasal lateral segment BL-WTd Wall thickness of basal lateral segment indiastole BL-WTs Wall thickness of basal lateral segment in systole BS-%WT Percentile wall thickening of basal septal segment BS-WTd Wallthickness of basal septal segment in diastole BS-WTs Wall thickness ofbasal septal segment in systole ML-% WT Percentile wall thickening ofmid lateral segment ML-WTd Wall thickness of mid lateral segment indiastole ML-WTs Wall thickness of mid lateral segment in systole MS-% WTPercentile wall thickening of mid septal segment MS-WTd Wall thicknessof mid septal segment in diastole MS-WTs Wall thickness of mid septalsegment in systole Global % WT Global percentual Wall Thickening of theLeft Ventricle AoV Vmean Aortic Valve mean Velocity AoV Pmean AorticValve mean Pressure gradient AoV VTI Aortic Valve Velocity Time IntegralAVA Vmax Aortic Valve Area (measured by max Vel.) AVA VTI Aortic valveArea (measured by Velocity Time Integral) AVAi Vmax Aortic Valve Area(measured by maximum Velocity) indexed to Body Surface Area AVAi VTIAortic valve Area (measured by Velocity Time Integral) indexed to BodySurface Area ivrt IsoVolumic Relaxation Time LV GLS (A4C) LeftVentricular Global Longitudinal Strain measured in A4C LV GLS (A2C) LeftVentricular Global Longitudinal Strain measured in A2C LV GLS (A3C) LeftVentricular Global Longitudinal Strain measured in A3C LV GLS LeftVentricular Global Longitudinal Strain (mean)

Referring again to FIG. 4A, in one embodiment, the echo workflow engine12 maintains measurement confidence scores corresponding to the measuredleft/right ventricles, left/right atriums, LVOTs, pericardiums andmeasured velocities. The echo workflow engine 12 filters out echo imageshaving measurement confidence scores that fail to meet a threshold,i.e., low measurement confidence scores (block 442).

Measurement of cardiac features continues with calculating longitudinalstrain graphs using the annotations generated by the CNNs (block 444).Thereafter, a fifth CNN is optionally used to detect pericardialeffusion 446.

FIG. 11 is a diagram graphically illustrating measurements of globallongitudinal strain that were automatically generated from theannotations of cardiac chambers in 2D images.

Referring again to FIG. 4A, for all remaining non-filtered out data, theecho workflow engine 12 selects as best measurement data themeasurements associated with cardiac chambers with the largest volumes,and saves with the best measurement data, image location,classification, annotation and other measurement data associated withthe best measurements (block 447).

FIG. 12A is a diagram graphically illustrating an example set of bestmeasurement data 1200 based on largest volume cardiac chambers and thesaving of the best measurement data 1200 to a repository, such asdatabase 16 of FIG. 1.

Referring again to FIG. 4A, the echo workflow engine 12 then generatesconclusions by inputting the best measurement data 1200 to a set ofrules based on international measurement guidelines to generateconclusions for medical decisions support (block 448).

FIG. 12B is a diagram graphically illustrating the input of normal LV EFmeasurements into a set of rules to determine a conclusion that thepatient has normal diastolic function, diastolic dysfunction, orindeterminate.

Referring again to FIG. 4A, after the conclusions are generated, areport is generated and output (FIG. 3 block 314), which may compriseblocks 450-456. Report generation may begin by the echo workflow engine12 outputting the best measurement data 1200 to a JSON file forflexibility of export to other applications (block 450).

FIG. 13 is a diagram graphically illustrating the output ofclassification, annotation and measurement data to an example JSON file.

Referring again to FIG. 4A, a lightweight browser-based user interface(UI) displayed showing a report that visualizes the best measurementdata 1200 from the JSON file and that is editable by a user (e.g.,doctor/technician) for human verification (block 452). As is well known,a lightweight web browser is a web browser that is optimize to reduceconsumption of system resources, particularly to minimize memoryfootprint, and by sacrificing some of the features of a mainstream webbrowser. In one embodiment, any edits made to the data are stored in thedatabase 16 and displayed in the UI.

In order to make clinically relevant suggestion to the user,measurements associated with the best measurement data 1200 areautomatically compared to current International guideline values and anyout of range values are highlighted for the user (block 454).

FIG. 14 is a diagram illustrating a portion of an example report showinghighlighting values that are outside the range of Internationalguidelines.

Referring again to FIG. 4A, the user is provided with an option ofgenerating a printable report showing an automated summary of MainFindings (i.e., a conclusion reached after examination) and underliningmeasurements of the patient's health (block 456).

FIG. 15 is a diagram illustrating a portion of an example report of MainFindings that may be printed and/or displayed by the user.

In one embodiment, the automated workflow of the echo workflow engine 12may end at block 456. However, in further aspects of the disclosedembodiments, the process may continue with advance functions, asdescribed below.

FIG. 4B is a flow diagram illustrating advanced functions of the echoworkflow engine. According to this embodiment, the echo workflow engine12, takes as inputs values of specific measurements that wereautomatically derived using machine learning (see block 310), andanalyzes the specific measurements to determine diseasediagnosis/prediction/prognosis versus both disease and matched controlsand normal patient controls (block 460). In one embodiment, the echoworkflow engine 12 may use the disease predictions to perform diagnosisof any combination of: cardiac amyloidosis, hypertrophic cardiomyopathy,restrictive pericarditis, cardiotoxity, early diastolic dysfunction andDoppler free diastolic dysfunction assessment (block 462). A prognosisin the form of an automated score may then be generated to predictmortality and hospitalizations in the future (block 464).

Echocardiography is key for the diagnosis of heart failure withpreserved ejection fraction (HFpEF). However, existing guidelines aremixed in their recommendations for echocardiogram criteria and none ofthe available guidelines have been validated against gold-standardinvasive hemodynamic measurements in HFpEF.

According to one embodiment, the echo workflow engine 12 furthergenerates a diagnostic score for understanding predictions (block 466).Using machine learning, the echo workflow engine 12 validates thediagnostic score against invasively measured pulmonary capillary wedgepressure (PCWP), and determines the prognostic utility of the score in alarge HFpEF cohort.

In one embodiment, the echo workflow engine 12, takes as the inputsvalues of specific measurements that were automatically derived usingmachine learning workflow, and analyzes the input values using an HFpEFalgorithm to compute the HFpEF diagnostic score.

Recognizing that hypertensive heart disease is the most common precursorto HFpEF and has overlapping echocardiogram characteristics with HFpEF,echocardiogram features of 233 patients with HFpEF (LVEF≥50%) wascompared to 273 hypertensive controls with normal ejection fraction butno heart failure. An agnostic model was developed using penalizedlogistic regression model and Classification and Regression Tree (CART)analysis. The association of the derived echocardiogram score withinvasively measured PCWP was investigated in a separate cohort of 96patients. The association of the score with the combined clinicaloutcomes of cardiovascular mortality of HF hospitalization wasinvestigated in 653 patients with HFpEF from the Americas echocardiogramsub study of the TOPCAT trial.

According to one embodiment, left ventricular ejection fraction(LVEF<60%), peak TR velocity (>2.3 m/s), relative wall thickness(RWT>0.39 mm), interventricular septal thickness (>12.2 mm) and E wave(>1 m/s) are selected as the most parsimonious combination of variablesto identify HFpEF from hypertensive controls. A weighted score (range0-9) based on these 5 echocardiogram variables had a combined area underthe curve of 0.9 for identifying HFpEF from hypertensive controls.

FIGS. 16A and 16B are diagrams showing a graph A plotting PCWP and HFpEFscores, and a graph B plotting CV mortality or hospitalization for HF,respectively. Graph A shows that in the independent cohort, the HFpEFscore was significantly associated with PCWP in patients with HFpEF(R²=0.22, P=0.034). Graph B shows that a one-point increase wasassociated with a 12% increase in risk (hazard ratio [HR] 1.12; 95% Cl1.02-1.23, P=0.015) for the combined outcome after multivariablecorrection.

According to the disclosed embodiments, the echocardiographic score candistinguish HFpEF from hypertensive controls and is associated withobjective measurements of severity and outcomes in HFpEF.

Neural Network Training

According to a further aspect of the disclosed embodiments, the ekoworkflow engine 12 incorporates a federated training platform foreffectively training the assortment of the neural networks employed bythe machine learning layer 200 (FIG. 2), as shown in FIG. 17.

FIG. 17 is a diagram illustrating a federated training platformassociated with the automated clinical workflow system. According to oneembodiment, the federated training platform 1900 comprises the ekoworkflow engine 1901 executing on one or more servers 1902 in the cloud.As described above, the eko workflow engine 1901 comprises multipleneural networks (NNs) 1908, which may include some combination of GANsand CNNs, for example. The servers 1902 and eko workflow engine 1901 arein network communication with remote computers 1903 a, 1903 b, 1903 n(collectively computers 1903) located on premises at respectivelaboratories 1904 (e.g., lab 1, lab2 . . . , lab N). Each of thelaboratories 1904 maintains image file archives comprising, for example,echocardiogram image files 1906 a, 1906 b . . . , 1906N (collectivelyecho image files 1906) of patient cohorts. For example, the laboratories1904 may comprise a hospital or clinical lab environment whereEchocardiography is performed as a diagnostic aid in cardiology for themorphological and functional assessment of the heart. When performingechocardiography and taking manual measurements, doctors typicallyselect a small subset of the available videos from the echo image files1906, and may only measure about two of the frames in those videos,which typically may have 70-100 image frames each.

To increase the accuracy of the neural networks comprising the ekoworkflow engine, it would be desirable to make use of the each lab'secho image files 1906 as training data for machine learning. However,some or all of the labs 1904 may treat the image file archives asproprietary (graphically illustrated by the firewall), and thus do notallow their echo image files 1906 to leave the premises, which means theecho image files 1906 are unavailable as a source of training data.

According to another aspect of the disclosed embodiments, the federatedtraining platform 1900 unlocks the proprietary echo image files 1906 ofthe separate laboratories 1904. This is done by downloading andinstalling lightweight clients and a set of NNs 1908 a, 1908 b, 1908 con computers 1903 a, 1903 b, 1903 c (collectively computers 1903) localto the respective labs 1904. More specifically, lightweight clientexecuting on computer 1903 a of a first lab (Lab 1) accesses the firstlab's echo image files 1906 a and uses those echo image files 1906 a totrain the NNs 1908 a and upload a first trained set of NNs back to theserver 1902 after training. The first set of trained NNs 1908 are thentrained at a second lab (e.g., lab 2) by downloading the lightweightclients and NNs 1908 b the computer 1903 b located at the second lab 2.The lightweight client executing on the computer 1903 b of the secondlab can then access the second lab's echo image files 1906 b and usethose echo image files 1906 b to continue to continue to train the NNs1908 b and to upload a second trained of NNs set back to the server1902. This process may continue until the NN's complete training at thelast lab N by the lightweight client executing on the computer 14N ofthe last lab N to access lab N's echo image files 1906N to train the NNsand to upload a final trained set of neural networks to the server 1902.Once uploaded to the server 1902 the final train set of neural networksare then used in analysis mode to automatically recognize and analyzeecho images in the patient studies of the respective labs 1904.

The federated training platform 1900 results in a highly trained set ofNNs 1908 that produce measurements and predictions with a higher degreeof accuracy. Another benefit is that federated training platform 1900unlocks and extracts value from the existing stores of echo data. Theecho image files 1906 from the laboratories 1904 previously representedvast numbers of echoes from past patient studies and clinical trialsthat sat unused and unavailable for machine learning purposes becausethe images are unstructured, views are un-labelled, and most of theimages were ignored. Through the federated training platform 1900, theseunused and unavailable echo images are now processed by the lightweightclient of the eko workflow engine to create labelled echo images thatare stored in structured image databases 1910 a, 1910 b . . . , 190N, ateach of the labs 1904, which is a necessary prerequisite for any machinelearning training or machine learning experiments performed on theimages. In one embodiment, the structured image databases remain locatedon premises at the individual labs 1904 to comply with the securityrequirements (of the labs 1904 and/or the eko workflow engine 1901).

Accordingly, the federated training platform 1900 provides access tostructured echo image databases 1910 in multiple lab locations to allowdistributed neural network training and validation of disease predictionacross multiple patient cohorts without the either the original echoimages files 1906 or the labelled echo image files in the structureddatabase 1910 ever having to leave the premise of the labs 1904.

As stated above, the segmentation CNNs may be trained from hand-labeledreal images or artificial images generated by general adversarialnetworks (GANs).

FIG. 18 is a diagram showing use of a GAN for creating imagesegmentation training data. A GAN is a class of neural networks thatuses two neural networks, pitting one against the other adversarially inorder to generate new, synthetic data that can pass for real data. Forexample, the GAN 1800 trained on a training set of hand-labeledsegmentation images 1802 can generate artificial segmentation imagessimilar to the training set.

The GAN 1800 comprises one neural network called a generator 1806 andanother neural network called a discriminator 1808. The generator 1806generates candidate artificial images, while the discriminator 1808evaluates the candidate images to determine statistically which of thecandidates belong to the training set and which do not. Thediscriminator 1808 is first trained based on samples from the trainingset 1802 until the discriminator 1808 achieves acceptable accuracy.

The generator 1806 is an inverse convolutional network that receives asinput a vector of random noise 1810 and upsamples the random noise 1810into an artificial image 1812. This generated artificial image 1812 isinput into the discriminator 1808 along with a stream of images takenfrom the training set 1802. The discriminator 1808 is a standardconvolutional network that receives both the real and fake images andclassifies/categorizes the artificial images 1812 using binomialclassifier labeling, such “Real” or “Artificial”. Optionally, thediscriminator may return probabilities, a number between 0 and 1, where1 may represent a prediction of authenticity and 0 representingartificial. The discriminator 1808 is in a feedback loop with thetraining set of images 1802, which is known, so that the discriminator1808 becomes more skilled at identifying artificial images. Thegenerator 1806 is in a feedback loop with the discriminator 1808 so thatthe generator 1806 produces better candidate images. Both the trainingset 1902 and the artificial images can be used as a final training set1904.

FIG. 19 is a diagram showing components of a GAN for creating viewclassification training data. GAN 1900 is similar to GAN 1800 in termsof structure and operation, except that the training 1902 set compriseshand-labeled real classification images and the final training set 1904includes the training set 1902 and artificial classification images.

A method and system for training neural networks of a software-basedautomatic clinical workflow that recognizes and analyzes both 2D andDoppler modality Echocardiographic images for automated measurements andthe diagnosis, prediction and prognosis of heart disease has beendisclosed. The present invention has been described in accordance withthe embodiments shown, and there could be variations to the embodiments,and any variations would be within the spirit and scope of the presentinvention. For example, the exemplary embodiment can be implementedusing hardware, software, a computer readable medium containing programinstructions, or a combination thereof. Software written according tothe present invention is to be either stored in some form ofcomputer-readable medium such as a memory, a hard disk, or anoptical-ROM and is to be executed by a processor. Accordingly, manymodifications may be made by one of ordinary skill in the art withoutdeparting from the spirit and scope of the appended claims.

We claim:
 1. A computer-implemented method for training neural networksof an automated workflow performed by a software component executing ona server, the server in network communication with remote computers atrespective laboratories, the respective laboratories maintaining imagefile archives comprising echocardiogram image files, the methodcomprising: downloading and installing a client and a set of neuralnetworks to a first remote computer of a first laboratory, the clientaccessing the echocardiogram image files of the first laboratory totrain the set of neural networks and to upload a first trained set ofneural networks to the server; downloading and installing the client andthe first trained set of neural networks to a second remote computer ofa second laboratory, the client accessing the echocardiogram image filesof the second laboratory to continue to train the first trained set ofneural networks and to upload a second trained set of neural networks tothe server; and continuing the process until the client and the secondtrained set of neural networks is downloaded and installed to a lastremote computer of a last laboratory, the client accessing theechocardiogram image files of the last laboratory to continue to trainthe second trained set of neural networks and to upload a final trainedset of neural networks to the server.
 2. The method of claim 1, whereinthe echocardiogram image file archives of one or more of thelaboratories contain un-labelled echocardiogram images, the methodfurther comprising: processing, by the client, the un-labeledechocardiogram images to create labelled echocardiogram images, andstoring the labeled echocardiogram images in a structured database priorto training.
 3. The method of claim 1, further comprising: storing thestructured database locally at the one or more laboratories to complywith security requirements.
 4. The method of claim 1, furthercomprising: storing the structured database remotely at the server. 5.The method of claim 1, further comprising: using both realechocardiogram images and artificial echocardiogram images as trainingdata.
 6. The method of claim 1, further comprising: using, by thesoftware component executing on the server, the final trained set ofneural networks in an analysis mode to automatically recognize andanalyze the echocardiogram images in the patient studies of therespective labs.
 7. The method of claim 6, wherein the softwarecomponent automatically recognizes and analyzes echocardiogram imagesby: receiving, from a memory, one of the patient studies comprising aplurality of echocardiogram images; separating the plurality ofechocardiogram (echo) images according to 2D images and Doppler modalityimages; classifying the 2D images by view type; extracting features ofthe Doppler modality images and using the extracted features to classifythe Doppler modality images by region; segmenting each classified 2Dview type to produce segmented 2D images; segmenting each classified theDoppler modality region to generate waveform traces to produce segmentedDoppler modality images; using both the segmented 2D images and thesegmented Doppler modality images to calculate for the patient studymeasurements of cardiac features for both left and right sides of theheart; generating conclusions by comparing the calculated measurementswith International cardiac guidelines; and outputting a reporthighlighting ones of the calculated measurements that fall outside ofthe International cardiac guidelines.
 8. The method of claim 7, furthercomprising: implementing the final trained set of neural networks in theanalysis mode as: a first neural network to classify the 2D images byview type; a second set of neural networks to both extract features fromDoppler modality images and to use the extracted features to classifythe Doppler modality images by region; a third set of neural networksfor each classified 2D view type in order to segment the cardiacchambers in the 2D images and produce segmented 2D images; and a fourthset of neural networks for each classified Doppler modality region inorder to segment Doppler modality images to generate waveform traces. 9.The method of claim 8, further comprising: including in the finaltrained set of neural networks a set of one or more prediction CNNs fordisease prediction and a set of one or more prognosis CNNs for diseaseprognosis.
 10. The method of claim 8, further comprising: implementingthe first neural network and the second set of neural networks asconvolutional neural networks (CNNs).
 11. The method of claim 10,further comprising: implementing the third set of neural networks asgenerative adversarial neural networks (GANs).
 12. The method of claim8, further comprising: using another set of the GANs at least in part togenerate artificial segmentation training data and classificationtraining data.
 13. A system, comprising: a server comprising a memoryand a processor coupled to the memory, the server in networkcommunication with remote computers at respective laboratories, therespective laboratories maintaining image file archives comprisingechocardiogram image files; and a workflow engine executed on the serverthat is configured to: download and install a client and a set of neuralnetworks to a first remote computer of a first laboratory, the clientaccessing the echocardiogram image files of the first laboratory totrain the set of neural networks and to upload a first trained set ofneural networks to the server; download and install the client and thefirst trained set of neural networks to a second remote computer of asecond laboratory, the client accessing the echocardiogram image filesof the second laboratory to continue to train the first trained set ofneural networks and to upload a second trained set of neural networks tothe server; and continuing the process until the client and the secondtrained set of neural networks is downloaded and installed to a lastremote computer of a last laboratory, the client accessing theechocardiogram image files of the last laboratory to continue to trainthe second trained set of neural networks and to upload a final trainedset of neural networks to the server.
 14. The system of claim 13,wherein the echocardiogram image file archives of one or more of thelaboratories contain un-labelled echocardiogram images, the systemfurther comprising: processing, by the client, the un-labeledechocardiogram images to create labelled echocardiogram images, andstoring the labeled echocardiogram images in a structured database priorto training.
 15. The system of claim 13, wherein the structured databaseis stored locally at the one or more laboratories to comply withsecurity requirements.
 16. The system of claim 13, wherein thestructured database is stored remotely at the server.
 17. The system ofclaim 13, wherein both real echocardiogram images and artificialechocardiogram images are used as training data.
 18. The system of claim13, wherein the final trained set of neural networks are used in ananalysis mode to automatically recognize and analyze the echocardiogramimages in the patient studies of the respective labs.
 19. The system ofclaim 18, wherein the software component automatically recognizes andanalyzes echocardiogram images by: receiving, from a memory, one of thepatient studies comprising a plurality of echocardiogram images;separating the plurality of echocardiogram (echo) images according to 2Dimages and Doppler modality images; classifying the 2D images by viewtype; extracting features of the Doppler modality images and using theextracted features to classify the Doppler modality images by region;segmenting each classified 2D view type to produce segmented 2D images;segmenting each classified the Doppler modality region to generatewaveform traces to produce segmented Doppler modality images; using boththe segmented 2D images and the segmented Doppler modality images tocalculate for the patient study measurements of cardiac features forboth left and right sides of the heart; generating conclusions bycomparing the calculated measurements with International cardiacguidelines; and outputting a report highlighting ones of the calculatedmeasurements that fall outside of the International cardiac guidelines.20. The system of claim 19, wherein the final trained set of neuralnetworks are implemented in the analysis mode as: a first neural networkto classify the 2D images by view type; a second set of neural networksto both extract features from Doppler modality images and to use theextracted features to classify the Doppler modality images by region; athird set of neural networks for each classified 2D view type in orderto segment the cardiac chambers in the 2D images and produce segmented2D images; and a fourth set of neural networks for each classifiedDoppler modality region in order to segment Doppler modality images togenerate waveform traces.
 21. The system of claim 20, wherein the finaltrained set of neural networks includes a set of one or more predictionCNNs for disease prediction and a set of one or more prognosis CNNs fordisease prognosis.
 22. The system of claim 20, wherein the first neuralnetwork and the second set of neural networks comprise convolutionalneural networks (CNNs).
 23. The system of claim 19, wherein the thirdset of neural networks comprise generative adversarial neural networks(GANs).
 24. The system of claim 20, wherein another set of the GANs isused at least in part to generate artificial segmentation training dataand classification training data.
 25. An executable software productstored on a non-transitory computer-readable medium containing programinstructions for training neural networks of an automated workflow, theprogram instructions for: downloading and installing a client and a setof neural networks to a first remote computer of a first laboratory, thelightweight client accessing the echocardiogram image files of the firstlaboratory to train the set of neural networks and to upload a firsttrained set of neural networks to the server; downloading and installingthe client and the first trained set of neural networks to a secondremote computer of a second laboratory, the lightweight client accessingthe echocardiogram image files of the second laboratory to continue totrain the first trained set of neural networks and to upload a secondtrained set of neural networks to the server; and continuing the processuntil the client and the second trained set of neural networks isdownloaded and installed to a last remote computer of a last laboratory,the lightweight client accessing the echocardiogram image files of thelast laboratory to continue to train the second trained set of neuralnetworks and to upload a final trained set of neural networks to theserver.